The invention disclosed herein relates generally to performing data transfer operations in a data storage system. More particularly, the present invention relates to facilitating data synchronization between a source and destination device in a storage operation system.
Performing data synchronization is an important task in any system that processes and manages data. Synchronization is particularly important when a data volume residing in one location in a system is to be replicated and maintained on another part of the system. Replicated data volumes may be used, for example, for backup repositories, data stores, or in synchronous networks which may utilize multiple workstations requiring identical data storage.
File replication may include continually capturing write activity on a source computer and transmitting this write activity from the source computer to a destination or target computer in real-time or near real-time. A first step in existing file replication systems, as illustrated in FIG. 1A, is a synchronization process to ensure that the source data 22 at a source storage device and the destination data 24 at a destination storage device are the same. That is, before a destination computer 28 may begin storing write activity associated with the source data 22 at a source computer 26, the system 20 needs to first ensure that the previously written source data 22 is stored at the destination computer 28.
Problems in existing synchronization processes may occur as a result of low or insufficient bandwidth in a network connection 30 over which the source and destination computers 26, 28 communicate. Insufficient bandwidth over the connection 30 ultimately causes bottlenecks and network congestion. For example, if the rate of change of data at the source computer 26 is greater than the bandwidth available on the network connection 30, data replication may not occur since data at the source computer 26 will continue to change at a faster rate than it can be updated at the destination computer 28. Therefore, the attempts to synchronize the source and destination computers 26, 28 may continue indefinitely without success and one set of data will always lag behind the other.
Additional synchronization problems may arise due to hardware failure. If either the source computer 26 or the destination computer 28 were to fail, become unavailable, or have a failure of one of its storage components, application data may still be generated without system 20 being able to replicate the data to the other storage device. Neither computers 26 or 28 possess means of tracking data changes during such a failure. Other possible sources of disruption of replication operations in existing systems may include disrupted storage paths, broken communication links or exceeding the storage capacity of a storage device.
Additionally, some existing synchronization systems maintain continuity across multiple storage volumes using a wholesale copy routine. Such a routine entails periodically copying the most or all contents of a storage volume across the network to replace all the previous replication data. A storage policy or network administrator may control the operations and determine the frequency of the storage operation. Copying the entire contents of a storage volume across a network to a replication storage volume may be inefficient and can overload the network between the source computer 26 and the destination computer 28. Copying the entire volume across the network connection 30 between the two computers causes the connection 30 to become congested and unavailable for other operations or to other resources, which may lead to hardware or software operation failure, over-utilization of storage and network resources and lost information. A replication operation as described above may also lack the capability to encrypt or secure data transmitted across the network connection 30. A replication operation that takes place over a public network, such as the Internet, or publicly accessible wide area network (“WAN”), can subject the data to corruption or theft.